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Abstract Body 


Background / Context: 

It is often the case that policy programs operate differently for different groups of people 
based on various baseline characteristics. Thus there is often interest in how programs affect 
different smaller subgroups within the overall population receiving an intervention (Bloom & 
Michalopoulos, 2010). These types of inquiries expand the question of “what works?” to “what 
works for whom?” In addition to studying impact variation at the individual level, the setting in 
which a program is implemented is also critical to consider when understanding why a program 
is or is not effective (Boruch, 2005). These types of analyses ask the question “what works best 
in which contexts?” Setting-level moderators can exist at different levels, including the family, 
economic context, locale (urban/rural), or the prevalence of existing community services. 

In 2007, the Center for Economic Opportunity (CEO) in the Mayor’s Office of the City of 
New York mounted the first holistic Conditional Cash Transfer (CCT) initiative in an 
economically advanced, services rich jurisdiction. The initiative is known as Opportunity NYC- 
Family Rewards (henceforth “Family Rewards”). CCTs offer cash assistance to needy 
households, but condition these offers on families’ making prestipulated investments in their 
children’s education and health. Inspired by Mexico’s pioneering “Progresa” program, the 
effects of Family Rewards are being measured via a randomized control trial. Initial findings 
from the program’s early operating period show that the program reduced current poverty and 
hardship, increased receipt of preventive healthcare, and did not improve school outcomes 
overall for elementary, middle or high school students overall (Riccio et ah, 2010). 

Subgroup differences were found for educational impacts for high school students by 
proficiency level: While few impacts were observed for the full sample of high school students’ 
academic outcomes (i.e., school attendance, course credits, grade advancement, and standardized 
test results), there were no impacts for those who were not academically prepared at baseline and 
strong impacts among those who were better academically prepared at baseline (Riccio et ah, 
2010). In addition, impacts observed on adolescents’ time use activity patterns occurred only for 
adolescents who were proficient at baseline, and only for females (Morris et ah, 2012). This 
study builds on these findings by extending subgroup definitions to the contextual level, and 
builds on and informs ecological theory (Bronfenbrenner & Morris, 2006) by focusing on how 
key school and neighborhood characteristics may have moderated the effects of Family Rewards. 

Purpose / Objective / Research Question / Focus of Study: 

Aimed at low-income families in six of New York City’s highest-poverty communities, 
Family Rewards ties cash rewards to a set of activities and outcomes thought to be critical to 
families’ short- and long-term success in the areas of children’s education, family preventive 
health care, and parents’ employment (see Table 1 for a description of key program activities and 
incentive amounts). This study addresses the following research question: How do key 
characteristics of schools (size, resources, and perceptions of safety) and neighborhoods 
(neighborhood SES and racial composition) moderate the impacts of Family Rewards on 
children’s academic achievement, time -use, educational processes, parent financial investment in 
children’s education, and families’ poverty status and material hardship? 

Setting: 

The intervention was aimed at low-income families in six of New York City’s highest- 
poverty communities in the Bronx, Brooklyn and Manhattan. 
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Population / Participants / Subjects: 

Survey sample. The study involves approximately 3,750 families and their children selected 
for the 18-month survey, half of whom could receive the cash incentives if they meet the 
required conditions and half who were assigned to a control group. Eligible families had to have 
incomes at or below 130 percent of the federal poverty level and at least one child in the fourth, 
seventh, or ninth grade. These ages are widely believed to be at or near the start of critical 
educational transition years. A majority (81%) of the families were one-parent families at 
baseline. Over half of all families (57%) had only one or two children and 43 % of families had 
three or more children. Just under half (47%) of the families were Hispanic/Latino and most 
others (51%) were black, non-Hispanic/Latino. Just over half of parents (53%) were employed, 
with about 37 percent working full time. Half of the parents in the study had not completed high 
school and did not have a GED certificate, about a third (32 %) had only a high school diploma 
or GED certificate, and the rest of the parents (18 %) had an associate’s or bachelor’s degree. 
About 83% were U.S. citizens, and the rest (17%) were legal pennanent residents. 

Child and Family Embedded Study sample. A subset of 716 high school students and their 
parents were selected from the survey sample to participate in an additional 30-month survey to 
collect infonnation on child and family processes that may have been affected by the 
intervention. Information was collected as part of this child study on 5 1 1 of them, representing 
an effective response rate of 72 percent (77 percent of parents were contacted and gave consent 
for their children to participate; of these, 93 percent of children consented to participate and 
complete a survey). An analysis of the survey respondents showed negligible differences 
between the child study sample and the survey sample. This sample was very similar 
demographically to the survey sample. The mean age at the time of the random assignment was 
13.9 years, and at the time of the survey for this study was 16.6 years (SD= 0.83). Nearly all 
adolescents attended public schools, and only a little over a third were found to meet learning 
standards for proficiency in English Language Arts (ELA) and Math. 

Intervention / Program / Practice: 

ONYC-Family Rewards is a multi-pronged intervention in which cash incentives are used to 
promote the families’ participation in three key settings and systems -education, employment, 
and health care. ONYC-Family Rewards ties cash rewards to a pre-specified set of activities and 
outcomes in the areas of children’s education, family preventive health care, and parents’ 
employment (see Table 1 for program activities and incentive amounts). Reflecting the important 
role that parents can play in their children’s success in school, the incentives in this domain are 
intended to encourage parents to become more fully engaged with their children’s education. In this 
way, Family Rewards differs from school-based incentive programs that only offer rewards directly 
to students, largely bypassing their parents. Control group families were eligible for benefits 
generally serving low-income families, but did not receive any special services. 

To implement Family Rewards, Seedco, the main implementing agency, assembled a 
network of local organizations in the designated community districts. Called “Neighborhood 
Partner Organizations” (NPOs), these agencies recruited and enrolled eligible families into the 
research sample and served as the face of the program in the communities. Seedco verified that 
families earned rewards by using a combination of automated data from some City agencies and 
special “coupon books” forms submitted directly by participants. After verification, it initiated a 
process of transferring payments electronically into bank accounts that participants newly 
opened or into their existing accounts they attached to the program, or, if they preferred, onto 
stored value cards. The payments were made every two months and families could access the 
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money at any time through any ATM. Envisioned as an “incentives-only” intervention, the 
program model does not provide social services or case management. The program also does not 
provide any direct services, such as tutoring or skill training. However, it does include an 
information-and-referral component wherein the implementing agencies (Seedco and the NPOs) 
refer families (upon request) to other agencies in the community that provide relevant services. 

Research Design: 

To ensure that the program reached a broad cross-section of children, not just the most 
motivated and active, potentially eligible families living in the targeted communities were 
identified from school lists from the NYC Department of Education. MDRC split this list into 
random batches of smaller groups of families, which were distributed to the NPOs one batch at a 
time. Seedco and the NPOs then attempted to recruit all families in a given batch, through 
mailings, phone calls, and home visits, before requesting the next batch from MDRC. As long as 
one child in the family was in the fourth, seventh, or ninth grade, with proper documentation the 
whole family could enroll in the program, including siblings under the age of 1 8 and both 
parents, if they were married or in a legal domestic partnership. Several analyses comparing 
families who entered the sample with those who were not in the study suggest that, despite its 
voluntary nature, the recruited sample is not a distinctively more advantaged or less advantaged 
subset of the broader target population. Randomization took place at the family-level after 
families enrolled in the program, off-site at MDRC. MDRC used batch random assignment for 
this process, which ensures that program and control group status is assigned randomly within 
the group of processed enrollees and that there was no attempts to “game the system” by looking 
for a pattern in the random assignment and trying to circumvent it. Half of the applicant families 
were picked for Family Rewards and offered the incentives and half were assigned to a control 
group that was not offered the incentives. Using such a random process to allocate sample 
members to one group or the other helps ensure that the program effects estimated by the 
evaluation are truly a result of the intervention. 

Data Collection and Analysis: 

Data collection. Data will come from three primary sources. First, the evaluation in this study 
uses an extensive set of quantitative data. This information includes administrative records on school 
outcomes, employment, earnings, public health insurance, welfare and food stamp payments, and 
housing subsidies obtained from various New York City and New York State agencies; and one wave 
of a survey in which a subset of parents in the program and control groups are interviewed 1 8 months 
after treatment began. The parent-survey, administered by the Department of Information 
Resources (DIR), was administered via telephone using CATI and had a response rate of 82%. 

Second, the child and family embedded study survey was collected via surveys to assess 
three domains: (1) family context and decision-making (family climate, norms, and expectations; 
resource allocation; time use); (2) children’s motivations and perceived competencies (academic 
efficacy, outcome expectations, intrinsic/extrinsic motivation); and (3) children’s educational 
and mental health outcomes. Survey data was collect from parents and children and administered 
by the Department of Infonnation Resources (DIR) via telephone using a mixed-mode 
(CATI/Field) methodology. 

Third, data on baseline school characteristics will come from the NYC Department of 
Education’s division of Student Performance and Accountability which offers many data-based 
resources, including comprehensive annual school reports with information about student and 
teacher characteristics, test results, as well as graduation and dropout rates. Data will be retrieved 
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through the Research Alliance at NYU and will be linked to children using the DBN (district 
borough number). Data on families’ home neighborhood characteristics will come from the U.S. 
census tracts and will be geo-coded according to adolescents’ home census tracts using ArcGIS 
software (ESRI, 2011). Using geocoding procedures, each child will be linked to census tracts 
from their home neighborhoods, and corresponding census data will be used to identify their 
home neighborhood characteristics. 

Analysis. To test the hypotheses of paper 3, a set of hierarchical linear models (HLM) analyses 
will be used to understand whether and how levels of various school and neighborhood 
characteristics predict program impacts on specified adolescent academic and behavioral 
outcomes. Specifically, the following two-level model (with children nested in schools or 
neighborhoods) will be used to assess this (combined equation below): 

Y= uoo + uoi (environmental characteristics) + uw (treatment) + Un (environmental 
characteristic*treatment) + r 2 (treatment) + r 2 + fh.j (covarsj) + e 
where un represents the cross-level interaction with school or neighborhood characteristic and 
treatment status, or the estimate of if and how this characteristic moderates program impacts. 

Findings / Results: 

Preliminary analyses. FR was implemented in six community districts (CDs). Diversity exists 
both across CDs and within neighborhoods in each CD (e.g., in terms of racial composition, 
percent of residents on public assistance, residential stability, school quality). With geocoded 
data and more detailed information at the census tract level, such differences in characteristics 
can be gathered in further detail at the neighborhood level. Preliminary analyses show that there 
is some variation in impact estimates across the six CDs (e.g., in changes to adolescent time use; 
see Table 2) and that these estimates are significantly different from each other. A further 
analysis of the neighborhood characteristics associated with this variation will allow for a more 
nuanced understanding of what specific contextual circumstances led to these differences. This 
paper will look at three structural characteristics that have been predominantly focused on in 
adolescent neighborhood research: neighborhood SES, neighborhood residential stability, and 
neighborhood ethnic diversity. 

Preliminary analyses by school quality show some variation in impacts on educational on 
more process-oriented educational outcomes (e.g., adolescents attending the highest ranked 
schools are more likely to experience become academically-oriented as a result of FR. In 
addition, adolescents attending lowest-ranked schools are more likely to experience increases in 
academic motivation and adolescents attending schools ranked in the top third experiencing 
reductions in motivation; see Table 3). A further analysis of the school characteristics associated 
with this variation will allow for a more nuanced understanding of what specific contextual 
circumstances led to these differences. This paper will look at three dimensions of the school 
context that have been shown to influence adolescents’ academic achievement: school resources, 
school size, and perceptions of school safety. 

Conclusions: 

This study explores impact variation by school and neighborhood characteristics, exploring how 
academic outcomes, mediating educational processes, and family hardship vary by important 
theoretically and empirically proven neighborhood and school characteristics. 

This paper advances future efforts to expand and successfully implement conditional cash 
transfer programs. 
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Appendix B. Tables and Figures 

Table 1. Schedule of Education, Health, and Workforce Rewards 


Activity Reward Amount 

Education incentive! 

Elementary and middle school students 

Attends P5% of scheduled school days (discontinued after Tear 2) $25 per month 


Scores at proficiency level (or improve) on annual 
math and English Language Arts ( FT A ) tests 
Elementary school students 
Middle school students 


Parent discusses annual math and E1A test results with 
teachers i discontinued after Tear 2) 

High school students 

Attends 955» of scheduled school days 
Accumulates 1 1 course credits per sear 
Passes NYS Regents exams 
Takes PSAT 

Graduates from high school 
All grades 

Parent attends parent-teacher conference 

Child obtains library card (discontinued ater Tear 2) 


1300 per math rest: $300 per ELA test 
$350 per math test; 1350 per ELA test 

$25 for parents to download, print, and review 
results (up to 5 tunes per year) 

$25 (up to 2 tests per year) 


$50 per month 
$600 

$600 per exam passed (up to 5 exams) 
$50 for taking the test (up to 2 times) 
$-400 bonus 


$25 per conference (up to 2 ernes per sear) 
$50 once during program 


Parent reviews low-stakes interim tests 'discontinued qfter Tear 1) 


Health incentives 

Maintaining pubhc or private health insurance (discontinued qfter Tear 21 
For each parent covered 
If all children are covered 

Annual medical checkup 

Doctor-recommended follow-up visit t discontinued qfter Tear 2) 

Early-mterventiou evaluation far child under 30 mondis old. 
if advised by pediatrician 

Prevencve dental care (cleaning checkup) 


Workforce incentives 

Sustained full-tune eurolovment 

Education and training while employed at least 10 hours per week 
'employment requirement discontinued qfter Tear 2) 


Per month $20 (public); $50 (private) 
Per month $20 (public); $50 (private) 

$200 per family member (once per year) 

$100 per family member (once per year) 


$200 per child (once per year) 

$100 per family member (ooce pet year for 
children 1-5 years old; twice per year for 
family members 6 years of age or older) 


$150 per month 

Amount varies by length of course, up to a 
maximum of $3,000 over three years 
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Table 2. Impacts of Family Rewards on Select Adolescent Outcomes, by School Environment 


School Quality Rank 


Bottom third 

Middle third 

T op third 

Global F-test 

post-hoc 

Diff 

(Impact) 


Diff 

(Impact) 

F- or x stat p- 

■value 

ref= 
top third 

Time use activity pattern (°/o) 







Academic 

0.07 

0.10 

0.13 * 

0.52 

0.770 


Social 

-0.06 

-0.02 

-0.09 

1.16 

0.559 


TV/Computer 

0.04 

-0.11 

-0.16 * 

3.29 

0.193 


Maintenance/W ork 

-0.06 

0.03 

0.13 * 

4.27 

0.113 

lo-hi 

Motivation to Learn 







Extrinsic motivation 

0.13 

0.04 

-0.15 

1.95 

0.144 

lo-hi 

Intrinsic motivation 

0.21 

-0.03 

-0.22 

2.56 

0.078 

lo-hi 







lo-hi 

Identified motivation 

q 23 *** 

0.05 

-0.17 * 

5.46 

0.005 

med-hi 

Introjected motviation 

0.24 ** 

0.03 

-0.03 

1.73 

0.179 

lo-hi 

Academic Processes 







Mastery goal orientation 

0.02 

0.07 

-0.12 

1.01 

0.365 

med-hi 

Performance avoidant goal orientation 

0.12 

0.31 

-0.13 

1.38 

0.252 


School engagement 

0.07 

-0.09 

0.04 

0.65 

0.520 


Academic efficacy 

0.10 

0.10 

0.03 

0.09 

0.914 


Participation in organized activities 







School activity (%) 

-0.05 

0.18 ** 

-0.06 

4.23 

0.113 

med-hi 

Organized activity index 

-0.07 

0.10 

0.29 

0.45 

0.637 


„ Sample size (total=511) 

177 

152 

142 
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Table 3. Impacts of Family Rewards on Select Adolescent Outcomes, by Site 


Site 1 ; 

1 Site 2 

1 Site 3 I 

! Site 4 ( 

! Site 5 

1 Site 6 

Global F-te st 

post-hoc 



Diff 


Diff 


Diff 


Diff 


Diff 


Diff 

F- or y 2 stat 


ref= 


C ontrol 

(Impact) 


(Impact) 


(Impact) 


(Impact) 


(Impact) 


(Impact) 

p-value 

site 6 

Time use group membership (°/o) 

Academic 

0.16 

0.00 

0.20 

0.00 

0.20 

0.06 

0.26 

0.05 

0.18 

0.20 * 

0.21 

-0.02 

4.11 

0.534 

5-6 

Social 

0.17 

-0.15 * 

0.16 

0.04 

0.28 

-0.06 

0.17 

0.00 

0.16 

-0.06 

0.30 

-0.10 

2.11 

0.833 


TV/Computer 

0.32 

0.10 

0.27 

0.03 

0.31 

-0.12 

0.27 

0.00 

0.48 

-0.10 

0.29 

0.04 

5.68 

0.338 

3-6 

Maintenance/W ork 

0.36 

0.04 

0.36 

-0.12 

0.21 

0.12 

0.31 

-0.05 

0.19 

-0.04 

0.20 

0.08 

3.01 

0.698 


Academic outcomes 
















Attendance is 95% or higher, Year 2 

0.25 

0.03 

0.25 

0.03 

0.28 

0.18 * 

0.29 

0.03 

0.30 

0.16 

0.20 

0.08 

4.03 

0.545 

3-6 

Attempted 1 1 or more credit, Year 2 

o.so 

003 

0.92 

0.02 

0.84 

o.os 

0 79 

0.14 * 

0.84 

0.06 

0.91 

-0.04 

5.59 

0.348 

3-6 , . 4-6 

Earned 22 or more credits. Years 1 & 2 

0.53 

0.03 

0.61 

0.10 

0.49 

0.13 

0.48 

0.00 

0.53 

0.06 

0.48 

-0.01 

4.82 

0.438 

3-6 

Passed at least 1 Regents exam(%) 

0.49 

0.12 

0.66 

0.04 

0.60 

Oil 

0.67 

-0.06 

0.54 

0.16 

0.53 

0.04 

6.86 

0.231 

4-6 

Passed at least 2 Regents exam (%) 

0.33 

-0.01 

0.49 

0.02 

0.42 

0.12 

0.49 

-0.05 

0.40 

0.25 ** 

033 

0.04 

4.77 

0.445 


Behavioral outcomes 
















Aggressive behaviors (%) 

0.26 

-0.22 ** 

0.25 

-0.10 

0.17 

-0.02 

0.16 

0.06 

0.30 

-0.12 

0.25 

-0.11 

7.90 

0.162 

1-6, 4-6 

Substance use (%) 

0.43 

-0.41 *** 

0.30 

-0.12 

0.23 

-0.13 

0.26 

-0.11 

0.40 

-0.20 

0.39 

-0.17 * 

2.94 

0.709 


Number of friends using substances 

2.09 

-0.39 

2.05 

-0.17 

1.96 

-0.10 

1.89 

-0.12 

2.18 

-0.44 ** 

1.89 

-0.06 

3.31 

0.652 


Motivation to Learn 
















Extrinsic motivation 

3.08 

0.13 

3.10 

-0.21 

3.17 

0.12 

3.15 

0.02 

2.95 

0.36 ** 

3.08 

-0.05 

1.11 

0.352 

5-6 

Intrinsic motivation 

2.42 

0.63 *** 

2.48 

0.13 

2.58 

-0.32 

2.68 

-0.06 

2.40 

0.06 

2.58 

-0.24 

2.00 

0.078 

1-6 

Identified motivation 

3.69 

0.10 

3.71 

-0.04 

3.62 

-0.03 

3.67 

0.05 

3.51 

0.21 

3.63 

0.01 

0.47 

0.798 


Introj e cte d motviation 

2.91 

0.33 * 

3.12 

-0 09 

3.11 

-008 

3.21 

-0.19 

2.84 

0.34 * 

3.02 

0.10 

1.17 

0324 


Academic Processes 
















Mastery goal orientation 

4.63 

0.01 

4.43 

0.03 

4.43 

0.09 

4.45 

-0.12 

4.42 

-0.01 

4.48 

0.05 

0.49 

0.784 


Performance avoidant goal orientation 

2.91 

ooo 

3.17 

0.07 

3.16 

-030 

2.78 

0.52 * 

3.05 

-0.16 

2.72 

0.60 ** 

1.64 

0.149 

3-6, 5-6 

School engagement 

4.45 

-003 

4.38 

-0 09 

4.14 

0.17 

4.29 

0.05 

4.15 

0.15 

4.29 

0.04 

0.14 

0.982 


Academic efficacy 

2.68 

0.36 ** 

2.78 

0.20 

2.95 

-0.0S 

300 

-0.03 

2.84 

0.02 

2.93 

0.05 

1.04 

0.393 

1-6 

Participation in organized activities 

School activity (%) 

0.62 

0.24 * 

0.53 

-0.07 

0.57 

-0.04 

0.48 

0.07 

0.73 

-0.13 

0.57 

-0.14 

7.28 

0.200 

1-6 

Organized activity index 

2.95 

0.47 

2.15 

037 

2.48 

0.57 

2.08 

0.32 

2.91 

-0.53 

2.54 

-0.45 

0.53 

0.756 

1-6 

Sample size 

64 


1 1A 1 

1 26 1 

1 93 1 

1 Zi 1 

1 m 
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